Skip to content

fix(cuda.core): run cuMemPoolSetAccess in relaxed mode in PinnedMemoryResource - #3475

Draft
juenglin wants to merge 1 commit into
NVIDIA:mainfrom
juenglin:fix/pmr-global-capture
Draft

juenglin wants to merge 1 commit into
NVIDIA:mainfrom
juenglin:fix/pmr-global-capture

Conversation

@juenglin

Copy link
Copy Markdown
Contributor

closes #3451

Since #3405, PinnedMemoryResource.__init__ without options calls cuMemPoolSetAccess to grant the current device read/write access to the driver's default host pool. The CUDA driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. This breaks code that constructs PinnedMemoryResource() while another thread holds an active global-mode graph capture — including PyTorch's torch.cuda.graph, which uses global mode by default.

The fix introduces MP_set_access in _memory_pool.pyx, wrapping cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode / _exit_relaxed_capture_mode helpers. This is the same pattern already used for pool attribute reads and writes and trim() since #2834. _PMR_init calls the new helper instead of the bare with nogil: HANDLE_RETURN(...) block.

The with-options path (cuMemPoolCreate) is intentionally unaffected; that call remains refused under global capture as documented in the 1.3.0 notes.

Files changed:

  • cuda_core/cuda/core/_memory/_memory_pool.pxd — declare MP_set_access
  • cuda_core/cuda/core/_memory/_memory_pool.pyx — implement MP_set_access with relaxed-capture wrapper
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx — call MP_set_access; remove now-unused as_cu cimport; add one-sentence docstring note
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi — stub regenerated by pre-commit
  • cuda_core/tests/test_memory.py — add test_pmr_construction_during_capture_keeps_capture_valid (three capture combinations) and test_pmr_construction_restores_capture_mode, modelled on the existing test_mempool_calls_during_capture_keep_capture_valid and test_dmr_init_restores_capture_mode
  • cuda_core/docs/source/release/1.3.1-notes.rst — new release notes file with fix entry

Counterpart review (AGENTS.md dual-root requirement): This change is entirely in cuda_core/, which is single-sourced. The cuda_bindings_12 / cuda_bindings_13 roots are unaffected; cuMemPoolSetAccess wrappers there are unchanged and no behavioral change applies to them.

SYNCED blocks: No SYNCED blocks are touched by this diff.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…yResource

Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool.  The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread.  Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.

Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since NVIDIA#2834.
Call it from _PMR_init instead of the bare with-nogil block.

The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.

Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.

Closes NVIDIA#3451
@juenglin juenglin added this to the cuda.core 1.3.1 milestone Nov 16, 2026
@juenglin juenglin added bug Something isn't working P0 High priority - Must do! cuda.core Everything related to the cuda.core module to-be-backported Trigger the bot to raise a backport PR upon merge labels Nov 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do! to-be-backported Trigger the bot to raise a backport PR upon merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Constructing PinnedMemoryResource() invalidates a global-mode graph capture on another thread

1 participant